Learning to Race: Experiments with a Simulated Race Car

Experiments with a Simulated Race Car Larry D. Pyeatt Adele E. Howe Colorado State University Fort Collins, CO 80523 email: fpyeatt,[email protected] URL: http://www.cs.colostate.edu/~fpyeatt,howeg Abstract We have implemented a reinforcement learning architecture as the reactive component of a two layer control system for a simulated race car. We have found that separating the layers has expedited gradually improving competition and multagent interaction. We ran experiments to test the tuning, decomposition and coordination of the low level behaviors. We then extended our control system to allow passing of other cars and tested its ability to avoid collisions. The best design used reinforcement learning with separate networks for each behavior, coarse coded input and a simple rule based coordination mechanism. Introduction Autonomous agents require a mix of behaviors, i.e., responses to di erent stimuli. This is especially true in situations where there are other agents present or where the environment is otherwise nondeterministic. For an agent to be e ective in its environment, it must have a large repertoire of behaviors and must be able to coordinate the use of those behaviors e ectively. Reactive systems have been favored for applications requiring quick responses, such as robotic or process control applications(Arkin 1994). Unfortunately, it is di cult to tune, decompose and coordinate reactive behaviors while ensuring consistency. In this paper, we present a case study of designing a mostly reactive autonomous agent for learning to perform a task in a multi-agent environment. The agent controls a race car in the Robot Automobile Racing Simulator (RARS) system (Timin 1995). To be successful, the agent must quickly respond to its environment and must have mastered several skills: steering, accelerating, and passing. To address these requirements, our basic agent architecture (Figure 1) adopts the common two-layer control structure: low level behaviors (implemented Copyright c 1998, American Association for Articial Intelligence (www.aaai.org). All rights reserved. Partial State Vector Sensors Effectors Actions Environment